Estimating Heterogeneous Treatment Effects of a Fractions Tutor

In this project, we aim to estimate the effect of different teaching strategies in a tutoring system on student learning and how that effect varies across different groups of students. More specifically, we want to shed light on whether choosing exercise problems adaptively based on prior student performance is more effective at teaching elementary school students about fractions than non-adaptive problem selection strategies. To this end, we analyze data of a recent study performed by Doroudi et al. (2017) and estimate heterogeneous treatments effects (HTE) using basic Monte-Carlo estimation as well as 3 recently proposed state-of-the-art methods for HTE estimation. While the estimates suggest that there may be small positive effects in 5 grade students and students of certain schools, none of the detected effects is statistically significant. This result raises the question what effect sizes current methods are able to detect for a given number of samples and shows the need for further extensive experimental comparisons across a wide range of scenarios.